(1) Field of the Invention
The present invention relates to a nose-view monitoring apparatus designed to display a pickup image taken through a nose-view camera(s) and further to extract optical flow vectors from the pickup image.
(2) Description of Related Art
So far, there has been developed a technique in which an image pickup camera is located at a nose (front end) portion of a vehicle so as to pick up a side-view image (right- and/or left-side image of lateral regions) and each of the pickup image (so-called nose-view image) is displayed on a monitoring apparatus or the like for supplementing the occupant's unaided vision. In general, in such a technique, a monitoring apparatus, which is designed to display a pickup image, is also used as an in-vehicle monitor made to display a television image, a car navigation image (screen display with car navigation system) or the like, and a television image or a navigation image is displayed thereon when a vehicle is running while the display switching to a nose-view image automatically takes place when the vehicle approaches an intersection or a T junction and comes to a temporary stop.
In addition, Japanese Patent No. 3287817 discloses a vehicle camera apparatus attached to a front portion of a vehicle to take a side-view image to be displayed on a monitoring apparatus (display unit) in the interior of the vehicle wherein the apparatus is automatically switched on in accordance with a speed or deceleration of the vehicle before the stop of the vehicle or before the vehicle reaches a predetermined low-speed run. This arrangement enables the apparatus to be switched on to put a lateral pickup image from a nose portion on a display before a temporary stop of the vehicle, for example, when the vehicle approaches an intersection and the driver confirms the lateral safe conditions, which allows the driver to carry out the speedy and smooth lateral safe confirmations well in advance. On the other hand, the apparatus remains switched off in a case in which the vehicle is in a slow-driving condition due to traffic congestion, which can prevent the interference with a television image, a navigation image or the like.
Moreover, with respect to a pickup image to be taken through such a monitoring apparatus, there has been developed a technique which is for detecting a mobile object existing in the pickup image by using an optical flow. In this case, the optical flow signifies a two-dimensional velocity vector field on an image, i.e., an apparent velocity field of a moving body in a moving image. In such a technique, for example, a point recognizable as the same object is set (detected through arithmetic operations) as a feature point between two consecutive images picked up on a predetermined period so that a movement (moving direction and moving distance) of this feature point is calculated as a vector (this vector is an optical flow vector and is equally referred to simply as a flow vector and, sometimes, it is referred to as an optical flow). Moreover, when the feature point and the flow vector are calculated throughout the entire region within the pickup image, the information such as position of a mobile object and moving direction in the image are recognizable.
Furthermore, Japanese Patent No. 3398934 discloses a technique of conducting arithmetic operations for calculating an optical flow of a pickup image in a vehicle driving direction (forward direction which is a traveling direction of the vehicle) except a region corresponding to a scene outside a road in the image. Concretely, the calculation of the optical flow is made with respect to only a region including a lower portion surrounded by straight lines drawn from an infinite-point on an image to both lower corner portions of a screen and a peripheral portion of the infinite-point. This can reduce the arithmetic throughput and shorten the processing time for an increase in speed, in comparison with a case of the calculation of an optical flow throughout the entire region of the image.
Meanwhile, for recognizing an mobile object such as a vehicle approaching the own vehicle on the basis of a nose-view image taken in a lateral field of view with respect to the vehicle by using an optical flow, if the vehicle is in a stopping state, no flow vector appears on an object (for example, the background portion in an image, such as buildings, street trees) which does not vary the relative position with respect to the vehicle, while a flow vector is generated and develops on only a body which moves actually. This signifies that the mobile object exists in the flow vector developing place, thus enabling the accurate detection of the mobile object on the basis of the presence or absence of the flow vector.
However, in general, a situation requiring a nose-view image is a case of the entry into blind intersections or T junctions and, in this situation, in most cases the vehicle driver slowly advances the vehicle while seeing the nose-view image for the lateral safe confirmation. Thus, in a state where the vehicle is in movement, the angle (composition) itself of the nose-view image moves, thereby creating a problem in that a flow vector develops even on an object such as a building or street tree which constitutes the background in the image and which does not move actually and, hence, the background is detected as a mobile object. Moreover, difficulty is experienced in extracting/detecting only a mobile object approaching the own vehicle from an image taken in a situation that the angle itself varies.
In addition, in a case of carrying forward a vehicle in a state where a steering wheel is turned, the entire nose-view image moves in accordance with a turning radius or turning speed of the vehicle. That is, there exists a problem in that a flow vector develops on the entire image due to the movement of the image pickup camera itself, which causes an optical flow developing due to the movement of the own vehicle and an optical flow generated by an actual mobile object to be detected in a state mixed so that difficulty is encountered in extracting (detecting) these optical flows separately.
For solving this problem, for example, Japanese Patent 3239521 discloses a technique of calculating an apparent optical flow generated by a motion of the own vehicle in a mobile object recognition apparatus designed to calculate an optical flow on the basis of a front-view image relative to a vehicle. In this technique, the apparent optical flow is calculated on the basis of data on a moving direction of the own vehicle and a distance to an object related to the extraction of the optical flow. And in the technique, a mobile object is detected by compensating for the influence of the motion of the vehicle on the optical flow.
However, this technique requires not only the moving direction data such as vehicle traveling speed and yaw rate but also the distance to an object to be detected, for calculating the apparent optical flow, i.e., the optical flow developing due to the movement of the own vehicle, which leads to complicated arrangement and complicated arithmetic operations.
Moreover, for example, with respect to a pickup image taken through a camera made to pick up images in front or rear field of view, since an apparent speed of an object to be detected as a mobile object is calculated as a difference (relative speed) between a traveling speed of the own vehicle and a traveling speed of the object, the image pickup is made at a moving speed lower than the actual moving speed of the object and, in most case, the magnitude of the optical flow of the object, extracted from the pickup image, becomes relatively small.
On the other hand, in the case of a nose-view camera designed to take images in lateral field of view with respect to a vehicle, an object of image pickup is usually a traveling ordinary vehicle or the like in a state where the own vehicle is in a stopping or slow-moving condition. For this reason, the image pickup is made at a traveling speed almost equal to the actual traveling speed of the object (that is, since the traveling speed of the own vehicle is low, the apparent traveling speed of the object on the pickup image does not become considerably low), which frequently causes the extraction of a relatively large optical flow so that the arithmetic throughput increases in the arithmetic operation for the optical flow. Accordingly, the precise optical-flow based recognition of a mobile object such as a vehicle approaching the own vehicle in a nose-view image requires the employment of an arithmetic processing unit with a high processing ability (throughput), which leads to an increase in apparatus cost as a whole. Thus, for detecting a mobile object from a pickup image taken through a nose-view camera on the basis of an optical flow, one solution to a conventional problem involves the reduction of the arithmetic operations.